cuda: add bf16 triton gdn chunk path - #3

…g-runner op (cherry-picked from PR #3)

fe0f95c committed the GGUF engine-gate goldens without a test_op_parity runner,
turning build-test-cpu RED. The golden dir is owned by
tests/parity/test_qwen36_gguf_engine.cpp, not the op runner; PR #3 carries this
one-hunk fix — cherry-picked ahead of its GPU validation pass. Verified:
test_op_parity 4/4 (18 assertions) locally.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Codex:gpt-5 [Codex] (original hunk, PR #3); Claude Code:claude-opus-4-8 [ClaudeCode] (cherry-pick)

Move the completed MVP roadmap under .agents/completed, make roadmap_v1 the live work index, and carry the full v0 post-MVP queue forward as explicit tracks. Centralize feature specs and scoping artifacts, record the CUDA and M4 hosts, and encode the lifecycle in AGENTS.md.\n\nRecord the independent PR #3 do-not-merge verdict and its missing artifact/GPU validation requirements.\n\nFOLLOWING_AGENTS_PROTOCOL\nAssisted-by: Codex:gpt-5 [Codex]

Move the completed MVP roadmap under .agents/completed, make roadmap_v1 the live work index, and carry the full v0 post-MVP queue forward as explicit tracks. Centralize feature specs and scoping artifacts, record the CUDA and M4 hosts, and encode the lifecycle in AGENTS.md.

Record the independent PR #3 do-not-merge verdict and its missing artifact/GPU validation requirements.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5 [Codex]

Indent the automated commit heredoc so GitHub can parse the workflow, and trigger drift checks when CMake or the sync scripts change.

Keep PR #3 blocked on expected-artifact validation and CUDA gating; this change only restores the current workflow.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5 [Codex]

Indent the automated commit heredoc so GitHub can parse the workflow, and trigger drift checks when CMake or the sync scripts change.

Keep PR #3 blocked on expected-artifact validation and CUDA gating; this change only restores the current workflow.

FOLLOWING_AGENTS_PROTOCOL

Assisted-by: Codex:gpt-5 [Codex]

…ation pass for PR #3

Refresh the active-claim row per the claim protocol: new agent, remote dir
~/work/vllm.cpp-noPy (reused), branch merge-pr3-validation, scope = PR #3
files + the claim's matrix/ledger/roadmap rows. Validation list: clean
CUDA+Triton build of the merged tree, test_ops_gdn, both greedy gates,
same-binary scratch-pool A/B, AOT byte-reproducibility check.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]

…expr arm (nvcc #128-D unreachable-code error in the vendored build)

Found by the PR #3 GPU validation pass (CLAIM-PR3): the merged tree FAILED the
clean -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON build — exactly the flagged
never-compiled-TU risk. In the Tout=__nv_bfloat16 instantiation WITHOUT
VLLM_CPP_TRITON_CHUNKO_BF16 (i.e. today's vendored sm_121a build, which has no
bf16 chunk_o artifacts), the '#else return false;' branch made the shared
VT_CHECK tail unreachable: nvcc #128-D 'loop is not reachable' at the
VT_CHECK do-while, promoted to error by warnings-as-errors.

Fix is structural only — duplicate the VT_CHECK+return tail into the f32 and
compiled-bf16 arms so every instantiation's body ends reachable. No dispatch,
guard, or launch behavior change; the f32 path is byte-identical logic.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]

…ader + shared BF16 helpers (MODEL-TEXT-qwen3-qwen3-for-causal-lm)

W2 of the Qwen3-dense (`Qwen3ForCausalLM`, Qwen3-0.6B BF16) additive-model
bring-up: the safetensors WEIGHT LOADER. Host-only; the forward stays a
clear-throwing W3 stub (the W2 boundary).

Shared-helper extraction (SEAM GAP #3, behaviour-preserving):
- Move LoadBf16Direct/LoadBf16Transposed/LoadMergedBf16RawNK (+ MakeOwned/
  TransposeBf16) from the anon namespace of qwen3_5_dense_weights.cpp into a NEW
  shared header include/vllm/model_executor/models/dense_weight_loaders.h
  (vllm::dense_loaders, inline). The 27B loader using-imports them and forwards
  the public vllm::LoadMergedBf16RawNK. Only the diagnostic prefix generalized
  ("qwen3_5 dense:" -> "dense loader:"); loaded bytes byte-identical.

New loader (src/vllm/model_executor/models/qwen3_weights.cpp,
LoadQwen3ForCausalLMWeights) fills the fleshed-out Qwen3DenseWeights (qwen3.h).
Grounded in vLLM qwen3.py@e24d1b24: merged qkv_proj/gate_up_proj raw-NK
(packed_modules_mapping), per-head q/k RMSNorm, and tied lm_head — with
tie_word_embeddings=true the checkpoint's redundant lm_head.weight is SKIPPED
and lm_head aliases embed_tokens (mirrors AutoWeightsLoader
skip_prefixes=["lm_head."] + self.lm_head = self.model.embed_tokens).
tie_word_embeddings/attention_bias parsed from config.raw (deferred from W0).

Gates (all passed): dgx CUDA -Werror 0-warn; W2 load gate test_qwen3_load
1567/1567 on dgx (all 311 Qwen3-0.6B tensors mapped, shapes exact, tied lm_head
resolves, NO leftover); 27B test_qwen27_paged_engine 235/235 + 35B
test_qwen36_paged_engine 315/315 token-exact UNCHANGED (extraction
behaviour-preserving); compute-sanitizer memcheck 0 err / 0 leak on the load
path; full CPU ctest 125/125. Benchmark disposition NOT APPLICABLE (the SACRED
token-exact Qwen3-0.6B vs vLLM 0.25.0 oracle gate is W4).

Records: model-matrix W2 anchors, parity-ledger, state.md, coordination
CLAIM-MODEL-QWEN3-DENSE W2, README + docs/BENCHMARKS, spike work breakdown.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

…ry stub + reusable-piece refactors

First full-attention MoE bring-up (breadth-sweep model #1). W0 registers the
arch; W1 delivers the three behaviour-preserving refactors that make the two
already-done paths (Qwen3-dense attention + 35B MoE experts) reusable by a
full-attention MoE. No forward yet (W2 loader / W3 forward throw clear stubs).

W0 — config + registry stub:
- NEW include/vllm/model_executor/models/qwen3_moe.h: Qwen3MoeWeights (reuses
  Qwen3DenseAttnWeights for attention + MoeBlockWeights for experts; untied
  lm_head) + Qwen3MoeModel::Forward/ForwardDevice decls.
- NEW src/vllm/model_executor/models/qwen3_moe_registry.cpp:
  REGISTER_VLLM_MODEL(qwen3_moe, "Qwen3MoeForCausalLM") + full-attention-ONLY
  MakeQwen3MoeKVCache (one FullAttentionSpec group, NO MambaSpec — clone of
  MakeQwen3ForCausalLMKVCache) + ParseQwen3MoeConfig (validates num_experts/
  num_experts_per_tok/moe_intermediate_size; shared_expert_intermediate_size==0
  is valid) + is_dense_model=false; load throws W2, forward throws W3.
- CMakeLists.txt: add the TU. NO runner change (the shape-agnostic runner covers
  gdn_group_id_<0 by construction).

W1 — additivity-enabling refactors (all behaviour-preserving):
- #1 EXTRACT the file-static dense self-attention block + device glue
  (Dev/DBuf/pool policy/ResidentWeight[F32]/KvSlice/StepInputs/AttnBlock + the
  env-flag readers) VERBATIM from qwen3.cpp to NEW dense_attn_block.h
  (namespace vllm::dense_attn, inline). qwen3.cpp `using namespace dense_attn`;
  its dense-only MLP/decoder-layer/forward-body compose the imported pieces.
  Pure relocation => Qwen3-dense byte-identical.
- #2 EXPOSE the file-static bf16 MoeBlock cross-TU via NEW qwen3_5_moe_block.h
  RunMoeBlock -> MoeBlockOutput (defined in qwen3_5.cpp outside the anon-ns;
  releases the internal DBuf into a Pool-returning shared_ptr). 35B untouched.
- #3 no-shared-expert GUARD in MoeBlock: shared_expert_intermediate_size==0 =>
  skip SharedExpert + pass nullptr to vt::MoeCombine (mirrors vLLM
  Qwen3MoeSparseMoeBlock shared_expert=None). Inert for the 35B (size>0).

Grounding: vllm/model_executor/models/qwen3_moe.py @ e24d1b24 (Qwen3MoeForCausalLM
:541-657, Qwen3MoeSparseMoeBlock :130-251, Qwen3MoeAttention == qwen3.py
Qwen3Attention); registry.py:192.

Gates (dgx, canonical cutlass-4.5.0 build, one flock): CUDA -Werror 0 warnings /
0 errors. Regression UNCHANGED (behaviour-preserving proof): Qwen3-dense near-tie
0.6B 16/16 (strict 10 + nt 6, max 0 nats) + 4B 16/16 (strict 11 + nt 5, max 0.25
nats) [664 assertions]; 27B test_qwen27_paged_engine 235/235; 35B
test_qwen36_paged_engine 315/315 — all token-exact. CPU test_model_registry
189/189 (resolves Qwen3MoeForCausalLM + full-attn-only KV + is_dense_model==false;
arch count 3->4) + test_model_loader_gguf 3/3. compute-sanitizer memcheck 0
access errors on the 0.6B dense gate (extracted AttnBlock).

Additivity: 4 new files + 5 shared touches (qwen3.cpp/qwen3_5.cpp byte-identical
for existing models, CMake TU, 2 test files for the arch count). W2 bf16 loader
-> W3 forward -> W4 near-tie vs vLLM 0.25.0 -> W5 fast bf16 grouped-MoE GEMM remain.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

…lity threading, tactic registry, queried smem ceiling

Closes the four seam-gaps the breadth-sweep audit named in
.agents/specs/breadth-sweep-plan.md §A.2, so adding a CUDA architecture vLLM
supports becomes a table-row edit plus a tactic registration instead of a
scattered multi-file edit. Structural/mechanical only: behavior-preserving on
GB10 (sm_121a) by construction, no new kernel, no numerics change.

GAP #1 — per-arch build FEATURE TABLE. The four hardcoded
`MATCHES "12[01]a"` guards (VT_FP4_MMA_SM120A, the CUTLASS NVFP4/FP8 TUs, the
vendored Marlin MoE) were whole-list substring tests: they could not say WHICH
arch supports what, enabled every feature globally for archs that cannot run
them, and silently dropped fp4/fp8/Marlin for legitimate spellings such as
"121" or vLLM's own "12.1a". Replaced by VT_CUDA_FEATURE_TABLE in the new
cmake/CudaArchFeatures.cmake, resolved per arch and REPORTED at configure time.
`cuda_archs_loose_intersection()` is a 1:1 port of vllm/cmake/utils.cmake:376-485
@ e24d1b24; the per-feature `<F>_ARCHS` + `if(<F>_ARCHS)` idiom mirrors vLLM
CMakeLists.txt:949-953,963 (FP4_SM120), :775-787 (sm120 SCALED_MM), :556-558
(MARLIN). Recorded deviation: the intersection is post-filtered to archs
literally requested, because the `a` suffix is load-bearing for the fp4 MMA.

GAP #4 — capability threaded to the kernel layer. New cuda_device_caps.h caches
ONE probe (compute capability, opt-in smem ceiling, SM count, residency
attributes); cuda_backend.cu's registrar now uses it and the backend CARRIES the
capability via new vt::Backend::DeviceCapabilityMajor/Minor(). Mirrors
vllm/platforms/cuda.py::CudaPlatform.get_device_capability.

GAP #2 — runtime SM-dispatch seam. New cuda_arch_tactics.{h,cu}: a type-erased
per-family tactic registry with capability predicates and selection counters.
LaunchFp4Fp4 now takes the capability and dispatches through it. Exactly ONE
tactic is registered — the existing sm_12x native fp4 path — so on GB10 this
reduces to the old `#if VT_FP4_MMA_SM120A && NativeFp4MmaEnabled()` test: same
kernel, same grid, same stream. A future arch ADDS a tactic from its own TU;
the launcher is never edited. Shape mirrors FlashInfer's per-arch tactic
registry (fp4_gemm_cutlass_template_sm120.h:187-220).

GAP #3 — queried smem ceiling. cuda_paged_attn.cu's 100 KiB opt-in ceiling was
a comment assumption with nothing checking it; seven open-coded opt-in blocks
became SetDynamicSmemOptIn(), which enforces the cached, QUERIED
cudaDevAttrMaxSharedMemoryPerBlockOptin (101376 on GB10 — every existing tile
already fits, so this is behavior-preserving) and names the device on a
shortfall instead of failing opaquely in the driver. The d==256 shape gate is
deliberately untouched.

Positive signal (a passing gate does not prove a new path ran): the registry
counts selections/fallbacks and VT_ARCH_TACTIC_STATS=1 announces the chosen
tactic; new tests assert the counters MOVE and that the backend reports the
driver's capability.

This does NOT make any untested architecture supported. One tactic is
registered, no row moves off INVENTORIED, and a heterogeneous fat build still
needs per-source gencode narrowing plus per-arch kernel bodies — a
hardware-blocked kernel campaign, not an additive drop-in.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

…d, records-only)

Multimodal SPEED track (ROAD-V1-MM). Dispatched to profile the mm serving
step and close the top reachable lever, but the dgx GPU environment was NOT
available this pass: ran on dev box mudler-ubuntu-box with no NVIDIA GPU, no
CUDA toolkit, dgx unreachable, no vllm-oracle venv, and root disk 99% full.
No nsys profile, build, oracle A/B, or gate re-run was possible; NO number
produced, NO repo code touched, NO row advanced (all mm rows stay
PARTIAL/speed-pending).

Deliverable (records only):
- Verified from OUR source the exact per-token host round-trips of the eager
  mm decode loop (qwen3_5.cpp:6871-6895: host MRoPE build :6877, a redundant
  embed D2H->H2D round-trip :6884/:6886, full-vocab logits D2H :6893 + host
  VLArgMax :6894; Voxtral voxtral.cpp:425-442 identical) -- the concrete
  target of lever #2 (on-GPU sampling for the cheap-decode audio path / c2+).
- Corrected the spec's qwen3_5.cpp:6756-6780 decode-loop mis-citation (that
  range is the per-layer KV/GDN-state ALLOC loop, not the decode loop).
- Added multimodal-speed.md S8 with the paste-ready dgx handoff recipe
  (do audio our-side timing #4, then on-GPU argmax #2, then batched serving #3).

Records: specs/multimodal-speed.md (S3/S5 citation fixes + S8), parity-ledger,
state, coordination (CLAIM-MULTIMODAL-SPEED-ATTR), minimal honest README +
docs/BENCHMARKS touch. Six record checkers green (agent-record, doc-checkpoint,
readme-structure, model-checklist, env-doc, fusion-consistency). Not pushed.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

…ver #2)

Closes multimodal-speed.md §5 lever #2 on the two mm eager greedy decode loops,
correctness held bit-for-bit.

- VLGenerateCoreGdn (qwen3_5.cpp, the shared 27B image+video core) and
  VoxtralGenerateGreedy (voxtral.cpp) now run the greedy pick ON the GPU via
  vt::GreedyArgmax (device vocab reduction, download only the winning int64 id)
  instead of D2H-ing the full [1,vocab] f32 logits and scanning on the host. The
  host VLArgMax/ArgMax scans are REMOVED -> the device argmax is the ONLY greedy
  path. Lowest-index tie-break (torch.argmax) == the removed host scan, so the
  greedy token stream is byte-for-byte unchanged.
- The decode loop embeds the fed token ON DEVICE and hands it straight to the
  forward; the redundant embed D2H->H2D round-trip is gone (voxtral
  ForwardLastLogits refactored to take a device Tensor and return a device DBuf).

Grounded in our production sampler (src/vllm/v1/sample/sampler.cpp:315-318 ->
vt::GreedyArgmax; CUDA two-pass lowest-index-tie src/vt/cuda/cuda_sample.cu:83-215)
and vLLM's greedy sampler path vllm/v1/sample/sampler.py (torch.argmax, lowest
index tie) @ pin 555967922.

CORRECTNESS (dgx GB10 sm_121a, clean cutlass 4.5.0 + FA2 + Triton-AOT build;
goldens md5-identical before+after): 27B image test_qwen3_5_vl_e2e STRICT 32/32
(54/54), 27B video test_qwen3_5_vl_video_e2e STRICT 32/32 (27/27), 4B image
test_qwen3vl_e2e STRICT 32/32 (46/46, unchanged code), Voxtral audio
test_voxtral_e2e PASS 14/14 (reproduces near-tie seq 48/48, strict prefix 33/48).

SPEED (same-binary A/B, throwaway VT_MM_HOST_ARGMAX toggle, rep0 dropped): Voxtral
decode TPOT 61.85 ms (61.73-61.94) vs 62.08 ms host = ~0.4% win; 27B image decode
223.0 ms (221.7-225.2) vs 224.0 ms host = NEUTRAL (~222 ms weight-streaming floor,
at vLLM parity). Honest: the win is small - even at 3B the eager forward dominates;
the audio 1.52x gap vs vLLM's 40.8 ms graphed is eager per-step launch overhead
(lever #3, graphed decode, for which on-GPU sampling is now a prerequisite in
place). mm rows stay PARTIAL (speed-pending). benchmark_binding=false.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

… is now GRAPH-CAPTURABLE (routes through the production captured decode); token-exact 32/32 held, NEUTRAL at the 27B bandwidth floor

SCOPE (grounded, file:line): mm SERVING ingestion is UNWIRED (protocol.cpp:298
bare-string only; no image_url/audio_url; no engine multi_modal_data path); the
mm decode was single-sequence eager (VLGenerateCoreGdn qwen3_5.cpp:6724,
VoxtralGenerateGreedy voxtral.cpp:375); the production graphed captured decode
Qwen3_5DenseDecodeGraph (qwen3_5_dense.h:314) already existed for the 27B-dense
family but the mm path bypassed it; Voxtral's Llama stack has NO decode-graph class.

BRICK: VLGenerateCoreGdn's pure-decode loop (shared 27B image+video) now runs each
step through Qwen3_5DenseDecodeGraph::Step (a per-generate instance, max_num_reqs=1)
instead of the eager DenseForwardLayers(...,&mrope_dec) — the mm decode step is now
graph-capturable, closing the un-graphed-eager-loop structural gap. Single-seq =>
B=1, PadToCaptureSize(1,1)=1 => S==B==1, the bit-identical-rebuild case. During
decode every position is text with MRoPE 3 axes equal ({p,p,p}) => MRoPE degenerates
to 1-D RoPE at p; Step applies device 1-D RoPE from positions={p} (p=abs_idx+delta),
reproducing the eager mrope angle. The eager path is preserved behind
VT_MM_DECODE_EAGER=1 (default = graph). One src file touched.

Grounded 1:1 in vLLM: the generic decode cudagraph dispatcher over mm requests with
the encoder kept eager (compile_mm_encoder:False) + EncoderCacheManager
(vllm/v1/core/encoder_cache_manager.py:17) @ pin 555967922.

CORRECTNESS (dgx GB10 sm_121a, RED line HELD, token-exact, proven-to-run): clean
build of bd3e15e, cutlass 4.5.0 + FA2 + Triton-AOT arch 121a, GPU under flock,
golden md5 3bc5f231 unchanged before+after. Proof-of-run: VT_DECODE_GRAPH_STATS
printed captured S=1 (real B=1) + 30 replays on each gate. 27B image
test_qwen3_5_vl_e2e STRICT 32/32 (54/54); 27B video test_qwen3_5_vl_video_e2e
STRICT 32/32 (27/27, teacher-forced gap 0 nats).

SPEED (same-binary A/B, throwaway VT_MM_DECODE_EAGER toggle, 4 reps/mode one load,
rep0 dropped, tpot=gen32_wall/31): graphed 232.5 ms/tok (231.8-233.9) vs eager
233.4 (233.35-233.5) = -0.9 ms/tok (~0.4%, graphed faster) = NEUTRAL at the ~222 ms
weight-streaming floor (the ~1 ms/tok launch overhead is hidden, as multimodal-speed
§8 predicted). Value is STRUCTURAL: decode is now graph-capturable (prerequisite for
batched c2+ and the audio launch-overhead win). mm rows stay PARTIAL (speed-pending).

W-plan (multimodal-speed.md §9.5): W1 Voxtral decode-graph = the audio 1.52x
gap-closer (3B decode is NOT bandwidth-floored); W2 batched multi-seq (c2+); W3
image_url/audio_url serving ingestion.

benchmark_binding=false. Additive routing (eager fallback retained), no new kernel.
No mm row advances to DONE.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 4.8 (claude-opus-4-8) via Claude Code

… decode GRAPH-CAPTURABLE; real non-overlapping win (60.94 vs 61.71 ms/tok) that NARROWS but does not close the 1.52x gap

Base: local main e2b18fc (the lever #3 FIRST-BRICK HEAD). Closes
multimodal-speed.md §9.5 W1.

SCOPE: Voxtral's Mistral/Llama text stack was the ONLY mm text stack with NO
decode-graph class (Qwen3.5-dense/MoE/DeepSeek all had one), so its audio decode
ran an eager per-step forward. W1 builds that class.

BRICK: new VoxtralDecodeGraph (voxtral.{h,cpp}) — the Voxtral-text sibling of
Qwen3MoeDecodeGraph (Qwen3-Coder): pure full-attention over the SAME
dense_attn::AttnBlock + vt::PagedAttention stack Voxtral uses (no GDN), with the
SAME cold->warm->replay state machine, padded-batch capture set
(decode_graph_sizes.h) and persistent fixed-address host inputs + persistent
embed/logits buffers. VoxtralGenerateGreedy's pure-decode loop now runs
VoxtralDecodeGraph::Step — the captured region is the EXACT ForwardLastLogits op
sequence the eager decode ran (embed kept OUTSIDE the capture via VoxtralEmbedInto;
S==B==1 => bit-identical rebuild). Eager path preserved behind VT_MM_DECODE_EAGER=1
(default = graph; parity-enabler-as-default). Two files (driver TU + header).

Grounded 1:1 in vLLM: the generic decode cudagraph dispatch —
gpu_model_runner.py::GPUModelRunner (_dummy_run warm-up then capture_model) +
compilation/cuda_graph.py (pad-to-nearest-captured-size) @ pin 555967922. Capture
safety with growing seq_len: hd-128 GQA-32/8 paged decode is the SAME path the
already-gated Qwen3-Coder decode graph captures (host max_seq_len only sizes the
split grid; geometry from DEVICE seq_lens — cuda_flash_attn_fa2.cu:23-31).

CORRECTNESS (dgx GB10 sm_121a, RED line HELD, token-exact, proven-to-run): clean
build cutlass 4.5.0 + FA2 + Triton-AOT arch 121a, GPU under flock /tmp/gpu; golden
md5 UNCHANGED before+after (voxtral_golden.json 8ab87b7e, voxtral_neartie.json
3d199c2d). Proof-of-run: VT_DECODE_GRAPH_STATS printed captured S=1 (real B=1) + 46
replays. test_voxtral_e2e PASS 14/14 (reproduces the committed near-tie seq 48/48,
strict prefix 33/48), held on all 12 A/B runs.

SPEED (same-binary A/B, throwaway VT_MM_DECODE_EAGER toggle, 6 reps/mode, rep0
dropped, steady-state TPOT excl. 2 cold+warm steps): graphed 60.94 ms/tok
(60.79-61.07) vs eager 61.71 (61.57-61.88) = -0.77 ms/tok (~1.25%, NON-OVERLAPPING
bands) — a real clean win, but it NARROWS the audio gap 1.52x->1.49x vs vLLM 0.25.0
graphed 40.8 ms and does NOT close it.

HONEST: the removable per-step launch overhead was only ~1.25% of TPOT, so the
~20 ms/tok residual is per-step COMPUTE/kernel efficiency (vLLM's torch.compile-
fused + graphed decode), NOT launch overhead — this REFINES the §9.5 hypothesis
that W1 is "the 1.52x gap-closer". Closing the audio gap needs a decode-kernel
nsys/port pass and/or batched c2+ (W2). STRUCTURAL value: Voxtral now has a
decode-graph class (prerequisite for batched multi-seq mm decode).

benchmark_binding=false. Additive class + eager fallback => text/other-model
forwards byte-identical by construction; no new kernel. mm rows stay PARTIAL
(speed-pending). No mm row advances to DONE.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 4.8 (claude-opus-4-8) via Claude Code

…0 ms/tok residual ATTRIBUTED to ONE kernel + a teacher-force-VALIDATED bf16 near-tie ceiling (records-only)

Closes multimodal-speed.md §10's follow-on (nsys our graphed Voxtral decode step
vs vLLM's, attribute the residual, port the divergent kernel 1:1 or characterize
the ceiling). CLAIM-MM-SPEED-DECODE-KERN. dgx GB10 sm_121a ~/work/mm-audio-kern,
cutlass 4.5.0 + FA2 + Triton-AOT arch 121a; ALL GPU under flock /tmp/gpu sole
owner. Teacher-force oracle ~/venvs/vllm-oracle-v0.25.0-stage (vLLM 0.25.0).

ATTRIBUTION (nsys cuda_gpu_kern_sum --cuda-graph-trace=node; decode kernels
isolated by instance count 1410 = 30 text layers x 47 steps): the WHOLE ~20 ms/tok
residual is the decode ATTENTION — the naive scalar vt::cuda::PagedAttentionKernel
(1410 @ 723 us/call = 21.7 ms/step, ~120x the KV memory floor), NOT the GEMMs
(cuBLAS gemvx, near-BW-floor, == vLLM's decode F.linear), NOT the lm_head cutlass
GEMM (~BW floor), NOT the RMSNorm/RoPE/SiLU glue (<0.3 ms/step, already
vt::FusedChain-folded).

LEVER (1:1 vLLM, already in-binary, gated off by a block_size quirk): Voxtral
(head_dim 128, GQA 32q/8kv bf16 causal) matches the DEFAULT-ON fa2_decode_qwen3
path (LaunchDecodeVarlenFA2Bf16 = our vendored flash_attn_varlen_func split-KV
decode, flash_fwd_splitkv) EXCEPT it needs block_size % 16 == 0, and the driver's
single KV block block_size = T0+max_new+8 = 444 is not a multiple of 16, so decode
fell to the scalar fallback. block_size/16 routes decode through FA2 (nsys of the
FA2 arm: flash_fwd_splitkv 1410 @ 18.5 us = 0.65 ms/step, 39x faster attention;
zero PagedAttentionKernel left in decode).

RESULT (same-binary A/B, throwaway block_size/16 + VT_FA2_DECODE_QWEN3 toggle,
steady-state, 4 reps rep0 dropped; instrumentation NOT committed): byte-exact NAIVE
59.4 ms/tok (59.25-59.53, repro 48/48) vs FA2 38.2 ms/tok (38.01-38.40, repro
18/48) = -21.2 ms/tok (~36%, NON-OVERLAPPING); 38.2 = 0.94x vLLM 0.25.0 graphed
40.8 ms — BEATS parity (naive 1.46x).

CEILING (why it cannot ship byte-exact): FA2's f32 reduction-order change flips the
committed near-tie golden's SOLE greedy branch (pos 33 = 4-way EXACT tie, gap
0.000) -> repro 48->18 -> the repro==48 assertion FAILS. But the FA2 sequence is
FULLY VALID: teacher-forcing vLLM 0.25.0 on it (a3_voxtral_neartie_gate.py) reports
0 divergent positions, worst gap 0.0000 nats, RESULT PASS — every one of the 48 FA2
tokens IS vLLM's teacher-forced argmax (a different-but-equal greedy branch, not a
bug). bf16 near-tie / golden-pinning ceiling; every faster decode-attention kernel
(FA2 or the PagedAttentionDecodeOpt/Gqa warp-shuffle kernels) changes the reduction
order, so there is NO byte-exact faster kernel; the full -21.2 ms is irreducible
under the byte-exact gate while the golden is pinned to the scalar kernel.

RED line HELD: shipped byte-exact scalar path re-verified on a clean rebuild of
bbcaedd — test_voxtral_e2e 14/14 (strict prefix 33/48, near-tie seq 48/48, worst
gap 0.0), goldens md5 UNCHANGED (voxtral_golden.json 8ab87b7e..., voxtral_neartie
.json 3d199c2d..., before == after). RECORDS-ONLY: no code change (voxtral.cpp
reverted to byte-exact base); no new/modified kernel => no compute-sanitizer
surface. benchmark_binding=false.

Reachable follow-on (USER DECISION on the golden-change policy): regenerate
voxtral_neartie.json::our_tokens from the FA2 sequence (teacher-force already PROVEN
PASS) + land block_size/16 -> claims a validated ~36% audio-decode win that BEATS
vLLM and closes the last mm speed gap. mm rows stay PARTIAL (speed-pending). No mm
row advances to DONE. Not pushed.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 4.8 (claude-opus-4-8) via Claude Code

…ode; BEATS vLLM (0.97x), closes the LAST mm decode-speed gap

USER-APPROVED adoption of the multimodal-speed.md §11.5 follow-on. Routes
Voxtral text decode through the FA2 varlen split-KV kernel via a one-line KV
block_size rounding, converts the near-tie gate to the ratified distributional
form, and regenerates the near-tie reference. Base: local main 57df9a9 (the §11
records-only HEAD). Verified on dgx GB10 sm_121a (cutlass 4.5.0 + FA2 + Triton-AOT,
-Werror clean).

Code (one line): VoxtralGenerateGreedy rounds the single KV block_size UP to a
multiple of 16 (((T0+max_new+8+15)/16)*16), so the pure-decode attention routes
through LaunchDecodeVarlenFA2Bf16 instead of the scalar PagedAttentionKernel. The
fa2_decode_qwen3 dispatch (cuda_paged_attn.cu:2620-2628) requires block_size%16==0
(line 2621); Voxtral (head_dim 128, GQA 32q/8kv, bf16, causal) matches every other
clause. Seq still fits one block, slot==abs_idx unchanged (no re-indexing) — only
the decode-attention kernel changes. FA2-routing PROVEN (nsys --cuda-graph-trace):
flash_fwd_splitkv 1410 @ 18.5us + combine 1410 @ 3.1us, ZERO PagedAttentionKernel.

Gate: test_voxtral_e2e converted to the user-ratified near-tie DISTRIBUTIONAL form.
Binding correctness = the teacher-force PASS (result==PASS + n_divergent==0 +
over_band==0 + worst_gap<=0.5), KERNEL-INDEPENDENT (both scalar and FA2 branch PASS).
Strict prefix = token-exact vs vLLM greedy up to the first genuine bf16 exact tie:
FA2 takes the other side of the pos-18 2-way EXACT tie (24466 vs golden 1584,
identical logprob -1.9875) so its exact prefix is 18 (the scalar branch's was 33;
both teacher-force valid) — asserted >=18. The old repro==48 byte-match to the scalar
branch is downgraded to a determinism anchor, regenerated to the FA2 sequence.
voxtral_neartie.json md5 3d199c2d -> 937b9ad3; STRICT greedy golden
voxtral_golden.json 8ab87b7e UNCHANGED. Gate PASS 16/16.

Teacher-force validation (vLLM 0.25.0 on the FA2 sequence): 0 divergent positions,
worst gap 0.0000 nats, RESULT PASS — every FA2 token IS vLLM's teacher-forced argmax.

CUDA-graph capture safety (FA2 runs inside the captured VoxtralDecodeGraph): graph
captured S=1 + 46 replays (all 48 tokens valid); compute-sanitizer memcheck 0 errors
on the graphed-FA2-decode surface (text-only: prefill + captured FA2 decode, 20
replays, 22/22, exit 0); 3 e2e runs byte-identical. Capture-safe — ships as the
DEFAULT graph path (no eager-FA2 fallback needed).

Speed (same-binary A/B, throwaway timer not committed, VT_FA2_DECODE_QWEN3 toggle,
6 reps/mode rep0 dropped, steady-state): scalar 60.50 ms/tok (60.39-60.62) vs FA2
39.50 ms/tok (39.41-39.58) = -21.0 ms/tok (~35%, NON-OVERLAPPING) = 0.97x vs vLLM
0.25.0 graphed 40.8 ms — BEATS parity.

Audio DECODE is now correctness- AND speed-DONE. The umbrella MM row
(ENG-MM-AUDIO-E2E) stays ACTIVE/PARTIAL: audio TTFT (the 32-layer Whisper encoder)
is UNMEASURED our-side vs vLLM 43 ms, and batched c2+ / audio_url serving ingestion
remain (same structural gaps as image/video).

Records: multimodal-speed.md §12 + headline; engine/model/feature matrices + roadmap
+ completion-spec MM lines (by-key); README + docs/BENCHMARKS.md; parity-ledger,
state, coordination CLAIM (append-only). Additive/kernel-config only; kAttention and
all shared ops untouched (text/other-model forwards byte-identical by construction).

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

HIGH-priority feature-gap #3. Rows QUANT-AWQ + QUANT-GPTQ move
INVENTORIED -> ACTIVE (both community formats).

W0 (.agents/specs/awq-gptq-quant.md): whole-chain spike over vLLM's
AWQ + GPTQ path — config parse (auto_awq.py/auto_gptq.py), the packed
layout (pack axis, reverse-AWQ vs standard bit order, zeros/scales,
act-order g_idx), the CPU reference dequant, the Marlin repack+GEMM vLLM
runs on GB10 (rides the vendored NVFP4 Marlin family), the
choose_mp_linear_kernel selection, upstream tests, W0-W6 breakdown.

W1: standalone CPU INT4 unpack+dequant-to-bf16 primitive for both
formats (awq_gptq_dequant.{h,cpp}), mirroring vLLM 1:1:
- DequantAwq4ToBf16  <- awq_triton.py:11-105 (reverse order [0,4,1,5,2,6,3,7], (w-z)*s, N-packed)
- DequantGptq4ToBf16 <- qdq_4.cuh dequant_4bit_8_gptq + q_gemm.cu:201-202
  (standard order, K-packed, optional act-order g_idx, zero_offset v1=1/v2=0)
Payloads read as uint32 so a set MSB never corrupts a nibble. No
loader/model/kernel edit — a pure primitive like nvfp4_dequant.cpp.

Gate (tests/vllm/test_awq_gptq_dequant.cpp, ported from
tests/kernels/quantization/test_awq_triton.py awq_dequantize_torch):
hand-computed known packed int32 (independent arithmetic oracle; AWQ
col7=15 set-MSB sign-safety; GPTQ zero_offset v1 AND v2; act-order
g_idx group-selection) + randomized double-precision layout roundtrip
through an independent reference packer + argument-validation aborts.
6 cases / 4189 assertions PASS. CPU -Werror build 0 warnings.

Records same-change: quantization-matrix (QUANT-AWQ/QUANT-GPTQ rows),
roadmap HIGH-gap #3, docs/STATUS, docs/BENCHMARKS (NOT-APPLICABLE —
dequant primitive), coordination CLAIM-QUANT-AWQ-GPTQ, parity-ledger,
state. Record checkers rc=0. Does NOT touch MXFP4/NVFP4 or the shared
Marlin templates (W4 GPU compute is a named next brick).

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

The Laguna decode regression that hid for weeks: our bf16 M=1 decode GEMVs
ran cuBLAS gemvx<bf16,FLOAT> (f32 output, 204us o_proj) where vLLM runs the
IDENTICAL kernel as gemvx<bf16,bf16> (bf16 output, 139us). The OUTPUT dtype
selects the gemvx template; requestedAlgoCount=1 skips the algo search. It hid
because parity was "verified" CROSS-TOOL (our nsys vs vLLM's torch profiler),
which cannot compare in-graph template dtypes. Root cause is CALLER-driven:
vt::MatmulBT is dtype-faithful, and an f32-resident C/D layout buys the slow
template.

Prevention layer (CPU-only, doc + checker):

- scripts/check-gemv-invocation-consistency.py + mutation test. Two op-contract
  invariants over src/vt/cuda/cuda_matmul.cu: (A1) every cuBLASLt C/D (output)
  layout dtype comes from the dtype-faithful out_type variable, never a
  hardcoded CUDA_R_32F literal (f32-out stays legal when the caller asks — only
  HARDCODING is banned); (A2) every requestedAlgoCount is the named constant
  kGemvHeuristicAlgos behind the /*requestedAlgoCount=*/ marker, never a bare
  literal. Pure functions, mutation-tested (hardcoded lc literal FAILS, bare
  algo literal FAILS, HEAD PASSES). Wired into ci.yml next to the sibling
  consistency checkers.

- src/vt/cuda/cuda_matmul.cu: byte-exact named-constant refactor —
  `constexpr int kGemvHeuristicAlgos = 1;` + substitute at all 4 heuristic
  sites. Zero behavior change (same int value/type). CUDA build-verify OWED on
  dgx (no nvcc on the authoring box).

- check-runner-routing-consistency.py invariant (c): enriched the per-model
  f32-stream drift message to name the concrete cost (gemvx<bf16,FLOAT> at M=1)
  and cross-reference the new op-contract gate. Message-only; its mutation test
  still passes.

- AGENTS.md + .agents/parity-lever-protocol.md: the four-axis cuBLAS/kernel
  INVOCATION-parity checklist (output dtype selects the template; compute+scale
  type; entry point + algo policy; resolved template dtypes off the SAME trace)
  plus the hard rule that a cross-tool comparison can NEVER establish invocation
  parity. parity-lever-protocol.md carries it as MANDATORY lane #3.

docs/STATUS.md + docs/BENCHMARKS.md updated same-commit (the CI guard + the
owed CUDA build-verify).

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]